Buy me a DrinkEther Coffee (5$)
Chaos Engineering Exploration: Production Deployment & Chaos Testing
Published on:
2024-10-20
#azure #aks #kubernetes #chaos-engineering #istio #chaos-mesh #azure-key-vault #istio-sidecar-injector #azure-managed-grafana
Table of Contents
1. Azure Services and Production Deployment
A. Production Architecture
The production architecture of the application is shown in the image below:
B. Architecture Components
The following components are used in the production deployment:
- AKS: The Kubernetes cluster is deployed on Azure Kubernetes Service (AKS) with the Istio service mesh extension.
- Azure Key Vault: The Key Vault is used to store the asymmetric key pairs for encrypting and decrypting the secrets’ AES Keys that are hosted in the database.
- Azure Managed Prometheus/Grafana: The Prometheus and Grafana services are managed by azure and are used to monitor everything in the Azure resource group.
C. Updates to the Dev/Test Kubernetes Cluster
The Dev/Test cluster was updated to include the following components:
- Cert-Manager: The Cert-Manager is used to manage the certificates for the application providing HTTPS support for the frontend and backend(Django API) services.
- Chaos Mesh: The Chaos Mesh is used to inject chaos into the application and test its resilience.
- Serice Types: The Service (Django API & VueJS) Types are chnaged from NodePort to LoadBalancer.
- Istio Ingress Controller: The Istio Ingress Controller is used to route the traffic to the appropriate services as well as to provide HTTPS support.
2. Initiation to Chaos Testing
A. Chaos Principles
- Start by defining ‘steady state’ as some measurable output of a system that indicates normal Behavior.
- Hypothesize that this steady state will continue in both the control group and the experimental group.
- Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc.
- Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.
- Template Hypothesis: Under ______ circumstances, customers will still have a good time.
B. Chaos Experiments (Basic)
Experiment1: Random Pod Deletion
- Hypthesis: Under the circumstances where a random pod gets deleted, customers will still be able to use the application without disturbances.
- Details: In this situation, the steady state is the availability of the application as a whole. It is measurable via the number of requests that have failed. The hypothesis requires an experiment, but due to the nature of this project, we cannot create two target groups (control group & experimental group) for the experiments.
- Results: As for the result, we can see, in the image above, that the Django pod has just been recreated seconds ago after it has been deleted by Chaos Mesh.
- Analysis: From the figure above, we can observe that the random pod's deletion caused a number of request to go unanswered (). Furthermore, the first figure shows that there is only one Django pod, which means that the ReplicaSet was set to ensure the existence of one pod, and that is exactly what triggered the recreation of the pod the moment the previous one was killed by Chaos Mesh. By this logic, we can say for sure that the configuration of more than one replica (as was configured in the previous chapter) will ensure the steady state is unaffected, for the Django pods at the very least.
- Counterfactual: Given the complexity of this system, it is challenging yet feasible to construct a mental model of how it behaves under different failure conditions. This experiment prompts an important question: What if a different pod had been deleted instead? While the exact outcome would require running additional experiments to observe, we can hypothesize based on the current system architecture.
For instance, reviewing the list of active pods reveals a unique one: the 'k8s-manager-pod.' Unlike others, this pod is not managed by a ReplicaSet, having been created by the 'KubeManager' application to provide access to the persistent volumes (PV) holding user files. If Chaos Mesh were to randomly delete this pod, it would not be automatically recreated, leading to a critical failure. Access to the user files stored on the PV would be permanently lost unless manual intervention occurred to remediate the situation. - Remediation: To mitigate this risk, it would be essential to ensure that critical pods like 'k8s-manager-pod' are either managed by a ReplicaSet or have a well-defined recovery strategy, such as using higher-level controllers like StatefulSets or creating specific health checks and alerts to trigger immediate manual recovery. By introducing these changes, we can reduce the potential impact of unexpected deletions and ensure the system remains resilient even when critical pods are affected.
Experiment2: Random Pod Failure
- Hypthesis: Under the circumstances where a pod experiences failure (without deletion), the application will suffer some minimal disruption as failing requests will still be routed to the unhealthy pod for a brief time.
- Details: The steady state in this case is the system's ability to maintain a high availability of services while handling the failing pod. The metric of interest will be the number of unanswered or failed requests.
- Results: In the image above, we can observe that the pods remain but 4/5 of them are in a state of CrashLoopBackOff. As a result, a portion of the requests routed to these 4 Django pod replicas goes unanswered, leading to a small number of request to fail in the meantime Kubernetes stops routing to them.
- Analysis: As shown in images above, the injected failure caused a portion of the requests to fail due to the unhealthy pods being included in the load balancer pool. The absence of proper health checks meant the system kept routing requests to the failing pod to some point before reaching the CrashLoopBackOff state.
This highlights a significant vulnerability in the system, where failed requestscould have been avoided with proper health check mechanisms. - Counterfactual: In this experiment, the failure was isolated to pods withouthealth checks in place. However, what if the system had health checks configured? In this case, the failing pod would have been quickly removed from the loadbalancer's pool, preventing further disruptions. Additionally, if multiple replicasof the pod existed, the impact of this failure could have been minimized evenfurther. Implementing readiness and liveness probes would allow the system todetect pod failures and trigger remediation processes, such as restarting the podor removing it from the active pool.
- Remediation: To prevent this issue from occurring in the future, we should implement proper health checks (readiness and liveness probes) for all pods, ensuring that unhealthy pods are detected and either restarted or removed from the load balancer's routing pool. Furthermore, increasing pod replicas would provide additional redundancy, reducing the overall impact of a single pod failure.
Experiment3: 5-second delay introduction to 50% of the requests received by the Django-api service
- Hypthesis: Under the circumstances where a 5-second delay is introduced to 50% of the requests received by the Django-api service, the response times of the service will increase, potentially leading to a degradation in user experience.
- Details: The steady state is the application's ability to respond within acceptable time limits. The hypothesis will be validated by measuring the impact of the injected latency on response times and error rates.
- Results: The experiment demonstrates that the 5-second delay introduced to 50% of the requests resulted in a measurable increase in response times affecting the loading speed of some UI elements but no requests were dropped as displayed in the followong image. The overall user experience was degraded as expected, but the system remained operational and no critical failures occurred.
- Analysis: The increased response times for the affected requests confirm that the injected latency had the anticipated impact on performance. The system handled the increased latency gracefully, confirming that it operates within acceptable parameters even when experiencing such delays.
- Conclusion: The experiment successfully confirms the hypothesis by demonstrating how the system's response times are affected by latency injection. The results build confidence in the system’s behavior under these conditions, indicating that the service can handle the injected latency without critical failures.
- Counterfactual: If the injected delay were introduced to the ‘KubeManager‘ service instead of the ‘Django-api‘ service, the situation could change signifi- cantly. The ‘KubeManager‘ is responsible for managing Kubernetes resources, including access to persistent volumes. Introducing a delay in this critical service might lead to severe disruptions, such as prolonged unavailability of persistentstorage or delays in resource management operations. Such disruptions could im-pact multiple services relying on ‘KubeManager‘, potentially leading to cascadingfailures across the cluster. In this case, a dedicated chaos experiment would berequired to evaluate the impact and ensure that recovery mechanisms are in placeto handle potential issues arising from such delays.
3. Conclusion & Aspirations
In this post, we concluded our project by exploring the potential impact of chaos engineering on the production deployment of a Kubernetes application. We demonstrated how chaos engineering can be used to simulate and test the resilience of the application, ensuring that it can withstand various failure scenarios.
Finally, I would like to thank you for your time and attention. I hope these posts have been helpful in any way. If you have any questions or comments, please feel free to reach out to me. I am always happy to discuss any related topics with you.